An Analysis-By-Synthesis Approach to Multisensory Object Shape Perception

The world is multimodal.1 We sense our environments using inputs from multiple sensory modalities. Similarly, digital information is increasingly available through multiple media. In this extended abstract, we present a general computational framework for understanding multimodal learning and perception that builds on the analysis-by-synthesis approach [2, 3]. The analysis-by-synthesis approach makes use of generative models describing how causes in the environment give rise to percepts. It achieves an analysis of the input as a synthesis of these causes by inverting this generative model. This approach has been successfully applied in a wide range of fields from vision[2] to language[3]. Our framework is a natural extension of this approach to multimodal perception. From the perspective of Marr’s tri-level hypothesis2, our framework can be understood as a computational level analysis of multimodal perception. It offers a conceptual analysis of multimodal perception that enables a unified treatment of a diverse set of questions. As a proof of concept of our framework, we apply it to object shape perception via visual and haptic modalities, and show that it captures people’s behavior in this task with high accuracy.